|Poster:||MeditateOrDie||Date:||Jul 31, 2017 11:42am|
|Forum:||faqs||Subject:||Re: robots.txt no longer supported to exclude page from being crawled?|
I'm not aware of any change to that policy.
Note: Not all site indexers/crawlers will obey robots.txt
so if you don't want something accessible, configure
your servers to use permissions or other tricks to
prevent unauthorized access to places which
you'd prefer to protect.